Phase Transitions in Rate Distortion Theory and Deep Learning

Abstract Rate distortion theory is concerned with optimally encoding signals from a given signal class $$\mathcal {S}$$ S using budget of R bits, as $$R \rightarrow \infty $$ R → ∞ . We say that can be compressed at rate s if we achieve an error most {O}(R^{-s})$$ O ( - s ) for the class; supremal compression denoted by $$s^*(\mathcal {S})$$ ∗ Given fixed coding scheme, there usually are some elements higher than scheme; in this paper, study size set signals. show certain “nice” classes , phase transition occurs: construct probability measure $$\mathbb {P}$$ P on such every scheme {C}$$ C and any $$s > s^*(\mathcal > encoded forms -null-set. In particular, our results apply to all unit balls Besov Sobolev spaces embed compactly into $$L^2 (\varOmega )$$ L 2 Ω bounded Lipschitz domain $$\varOmega As application, several existing sharpness concerning function approximation deep neural networks fact generically sharp addition, provide quantitative non-asymptotic bounds random $$f\in \mathcal f ∈ within accuracy $$\varepsilon ε bits. This result subsequently applied problem approximately representing (quantized) network W nonzero weights. constants c C that, no matter what kind “learning” procedure used produce network, success above $$\min \big \{1, 2^{C\cdot \lceil \log _2 (1+W) \rceil ^2 - c\cdot \varepsilon ^{-1/s}} \}$$ min { 1 , · W ⌈ log + ⌉ c / }